Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1579 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.5418. Min Act: -0.0000

Data Index: 543140 (C4 (Web Text))

Max Activating Token Index: 84

Text #1

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.4002. Min Act: -0.0000

Data Index: 790435 (C4 (Web Text))

Max Activating Token Index: 916

Text #2

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.3073. Min Act: -0.0000

Data Index: 111189 (C4 (Web Text))

Max Activating Token Index: 83

Text #3

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.3623. Min Act: -0.0000

Data Index: 210033 (C4 (Web Text))

Max Activating Token Index: 564

Text #4

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.2425. Min Act: -0.0000

Data Index: 1327908 (C4 (Web Text))

Max Activating Token Index: 203

Text #5

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.2334. Min Act: -0.0000

Data Index: 752982 (C4 (Web Text))

Max Activating Token Index: 649

Text #6

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1236. Min Act: -0.0000

Data Index: 446597 (C4 (Web Text))

Max Activating Token Index: 725

Text #7

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.2215. Min Act: -0.0000

Data Index: 525385 (C4 (Web Text))

Max Activating Token Index: 339

Text #8

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1431. Min Act: -0.0000

Data Index: 425409 (C4 (Web Text))

Max Activating Token Index: 254

Text #9

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0949. Min Act: -0.0000

Data Index: 1318167 (C4 (Web Text))

Max Activating Token Index: 992

Text #10

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1934. Min Act: -0.0000

Data Index: 943687 (C4 (Web Text))

Max Activating Token Index: 143

Text #11

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1183. Min Act: -0.0000

Data Index: 935564 (C4 (Web Text))

Max Activating Token Index: 310

Text #12

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1876. Min Act: -0.0000

Data Index: 1166098 (C4 (Web Text))

Max Activating Token Index: 261

Text #13

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0881. Min Act: -0.0000

Data Index: 247044 (C4 (Web Text))

Max Activating Token Index: 861

Text #14

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0160. Min Act: -0.0000

Data Index: 1322379 (C4 (Web Text))

Max Activating Token Index: 207

Text #15

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0066. Min Act: -0.0000

Data Index: 398205 (C4 (Web Text))

Max Activating Token Index: 980

Text #16

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0324. Min Act: -0.0000

Data Index: 568122 (C4 (Web Text))

Max Activating Token Index: 455

Text #17

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.1258. Min Act: -0.0000

Data Index: 1083595 (C4 (Web Text))

Max Activating Token Index: 725

Text #18

Max Range: 1.5418. Min Range: -1.5418

Max Act: 1.0648. Min Act: -0.0000

Data Index: 1198243 (C4 (Web Text))

Max Activating Token Index: 835

Text #19

Max Range: 1.5418. Min Range: -1.5418

Max Act: 0.9923. Min Act: -0.0000

Data Index: 1159722 (C4 (Web Text))

Max Activating Token Index: 17