Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1047 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 2.0244. Min Range: -2.0244

Max Act: 2.0244. Min Act: -0.0000

Data Index: 37530 (C4 (Web Text))

Max Activating Token Index: 475

Text #1

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.6681. Min Act: -0.0000

Data Index: 375309 (C4 (Web Text))

Max Activating Token Index: 381

Text #2

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.4963. Min Act: -0.0000

Data Index: 1222282 (C4 (Web Text))

Max Activating Token Index: 165

Text #3

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.5891. Min Act: -0.0000

Data Index: 380650 (C4 (Web Text))

Max Activating Token Index: 161

Text #4

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.4644. Min Act: -0.0000

Data Index: 90169 (C4 (Web Text))

Max Activating Token Index: 1006

Text #5

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.4469. Min Act: -0.0000

Data Index: 1032235 (C4 (Web Text))

Max Activating Token Index: 656

Text #6

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.4809. Min Act: -0.0001

Data Index: 207012 (C4 (Web Text))

Max Activating Token Index: 636

Text #7

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.4495. Min Act: -0.0001

Data Index: 741088 (C4 (Web Text))

Max Activating Token Index: 937

Text #8

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3923. Min Act: -0.0000

Data Index: 583277 (C4 (Web Text))

Max Activating Token Index: 685

Text #9

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3064. Min Act: -0.0000

Data Index: 124426 (C4 (Web Text))

Max Activating Token Index: 204

Text #10

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3111. Min Act: -0.0000

Data Index: 1021071 (C4 (Web Text))

Max Activating Token Index: 826

Text #11

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.2971. Min Act: -0.0000

Data Index: 718838 (C4 (Web Text))

Max Activating Token Index: 483

Text #12

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3036. Min Act: -0.0000

Data Index: 73960 (C4 (Web Text))

Max Activating Token Index: 879

Text #13

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3347. Min Act: -0.0001

Data Index: 1001860 (C4 (Web Text))

Max Activating Token Index: 115

Text #14

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.3000. Min Act: -0.0000

Data Index: 542993 (C4 (Web Text))

Max Activating Token Index: 224

Text #15

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.1896. Min Act: -0.0001

Data Index: 616440 (C4 (Web Text))

Max Activating Token Index: 731

Text #16

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.2311. Min Act: -0.0001

Data Index: 339739 (C4 (Web Text))

Max Activating Token Index: 979

Text #17

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.2296. Min Act: -0.0001

Data Index: 822695 (C4 (Web Text))

Max Activating Token Index: 142

Text #18

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.1828. Min Act: -0.0000

Data Index: 847974 (C4 (Web Text))

Max Activating Token Index: 942

Text #19

Max Range: 2.0244. Min Range: -2.0244

Max Act: 1.2940. Min Act: -0.0001

Data Index: 408589 (C4 (Web Text))

Max Activating Token Index: 205