Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5240 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.3248. Min Range: -3.3248

Max Act: 3.3248. Min Act: -0.0000

Data Index: 823426 (C4 (Web Text))

Max Activating Token Index: 101

Text #1

Max Range: 3.3248. Min Range: -3.3248

Max Act: 3.3065. Min Act: -0.0001

Data Index: 1044728 (C4 (Web Text))

Max Activating Token Index: 14

Text #2

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.8858. Min Act: -0.0001

Data Index: 324767 (C4 (Web Text))

Max Activating Token Index: 538

Text #3

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.8672. Min Act: -0.0001

Data Index: 981154 (C4 (Web Text))

Max Activating Token Index: 437

Text #4

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.5216. Min Act: -0.0001

Data Index: 585017 (C4 (Web Text))

Max Activating Token Index: 49

Text #5

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.5591. Min Act: -0.0001

Data Index: 187520 (C4 (Web Text))

Max Activating Token Index: 210

Text #6

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.5417. Min Act: -0.0001

Data Index: 326162 (C4 (Web Text))

Max Activating Token Index: 650

Text #7

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.5171. Min Act: -0.0000

Data Index: 1394674 (Python Code)

Max Activating Token Index: 14

Text #8

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.4458. Min Act: -0.0000

Data Index: 1184998 (C4 (Web Text))

Max Activating Token Index: 182

Text #9

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.5485. Min Act: -0.0000

Data Index: 1601172 (Python Code)

Max Activating Token Index: 423

Text #10

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3257. Min Act: -0.0001

Data Index: 766409 (C4 (Web Text))

Max Activating Token Index: 64

Text #11

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3475. Min Act: -0.0001

Data Index: 1025392 (C4 (Web Text))

Max Activating Token Index: 1009

Text #12

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3394. Min Act: -0.0000

Data Index: 1640427 (Python Code)

Max Activating Token Index: 159

Text #13

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.4391. Min Act: -0.0000

Data Index: 1594784 (Python Code)

Max Activating Token Index: 234

Text #14

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.1928. Min Act: -0.0001

Data Index: 1296734 (C4 (Web Text))

Max Activating Token Index: 303

Text #15

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3112. Min Act: -0.0000

Data Index: 691786 (C4 (Web Text))

Max Activating Token Index: 128

Text #16

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3217. Min Act: -0.0000

Data Index: 1601794 (Python Code)

Max Activating Token Index: 327

Text #17

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3864. Min Act: -0.0000

Data Index: 1177389 (C4 (Web Text))

Max Activating Token Index: 687

Text #18

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.1301. Min Act: -0.0000

Data Index: 178996 (C4 (Web Text))

Max Activating Token Index: 987

Text #19

Max Range: 3.3248. Min Range: -3.3248

Max Act: 2.3030. Min Act: -0.0001

Data Index: 1075031 (C4 (Web Text))

Max Activating Token Index: 584