Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3871 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3259. Min Range: -2.3259

Max Act: 2.3259. Min Act: -0.0000

Data Index: 13369 (C4 (Web Text))

Max Activating Token Index: 937

Text #1

Max Range: 2.3259. Min Range: -2.3259

Max Act: 2.1155. Min Act: -0.0000

Data Index: 681726 (C4 (Web Text))

Max Activating Token Index: 147

Text #2

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.8901. Min Act: -0.0000

Data Index: 965404 (C4 (Web Text))

Max Activating Token Index: 640

Text #3

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.7413. Min Act: -0.0000

Data Index: 435525 (C4 (Web Text))

Max Activating Token Index: 576

Text #4

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.8294. Min Act: -0.0000

Data Index: 1020210 (C4 (Web Text))

Max Activating Token Index: 348

Text #5

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.6650. Min Act: -0.0000

Data Index: 1208240 (C4 (Web Text))

Max Activating Token Index: 905

Text #6

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.8173. Min Act: -0.0000

Data Index: 286244 (C4 (Web Text))

Max Activating Token Index: 770

Text #7

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.5597. Min Act: -0.0000

Data Index: 338215 (C4 (Web Text))

Max Activating Token Index: 493

Text #8

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.6465. Min Act: -0.0000

Data Index: 804966 (C4 (Web Text))

Max Activating Token Index: 681

Text #9

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4709. Min Act: -0.0000

Data Index: 422798 (C4 (Web Text))

Max Activating Token Index: 449

Text #10

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.3528. Min Act: -0.0000

Data Index: 525090 (C4 (Web Text))

Max Activating Token Index: 666

Text #11

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.5140. Min Act: -0.0000

Data Index: 532731 (C4 (Web Text))

Max Activating Token Index: 692

Text #12

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4150. Min Act: -0.0000

Data Index: 316218 (C4 (Web Text))

Max Activating Token Index: 675

Text #13

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4688. Min Act: -0.0000

Data Index: 506108 (C4 (Web Text))

Max Activating Token Index: 980

Text #14

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.3898. Min Act: -0.0000

Data Index: 1026521 (C4 (Web Text))

Max Activating Token Index: 148

Text #15

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4540. Min Act: -0.0000

Data Index: 858546 (C4 (Web Text))

Max Activating Token Index: 48

Text #16

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4551. Min Act: -0.0000

Data Index: 693635 (C4 (Web Text))

Max Activating Token Index: 26

Text #17

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.5297. Min Act: -0.0000

Data Index: 651167 (C4 (Web Text))

Max Activating Token Index: 950

Text #18

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4277. Min Act: -0.0000

Data Index: 347455 (C4 (Web Text))

Max Activating Token Index: 811

Text #19

Max Range: 2.3259. Min Range: -2.3259

Max Act: 1.4312. Min Act: -0.0000

Data Index: 713605 (C4 (Web Text))

Max Activating Token Index: 168