Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 941 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2941. Min Act: -0.0000

Data Index: 542310 (C4 (Web Text))

Max Activating Token Index: 388

Text #1

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2582. Min Act: -0.0001

Data Index: 66282 (C4 (Web Text))

Max Activating Token Index: 941

Text #2

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2670. Min Act: -0.0000

Data Index: 681177 (C4 (Web Text))

Max Activating Token Index: 811

Text #3

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2589. Min Act: -0.0000

Data Index: 217654 (C4 (Web Text))

Max Activating Token Index: 539

Text #4

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2334. Min Act: -0.0000

Data Index: 565722 (C4 (Web Text))

Max Activating Token Index: 861

Text #5

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2458. Min Act: -0.0000

Data Index: 656709 (C4 (Web Text))

Max Activating Token Index: 488

Text #6

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2396. Min Act: -0.0001

Data Index: 126500 (C4 (Web Text))

Max Activating Token Index: 306

Text #7

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2406. Min Act: -0.0000

Data Index: 214630 (C4 (Web Text))

Max Activating Token Index: 556

Text #8

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2461. Min Act: -0.0000

Data Index: 907933 (C4 (Web Text))

Max Activating Token Index: 513

Text #9

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2330. Min Act: -0.0000

Data Index: 469348 (C4 (Web Text))

Max Activating Token Index: 23

Text #10

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2341. Min Act: -0.0000

Data Index: 1094906 (C4 (Web Text))

Max Activating Token Index: 614

Text #11

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2222. Min Act: -0.0000

Data Index: 523949 (C4 (Web Text))

Max Activating Token Index: 544

Text #12

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2216. Min Act: -0.0000

Data Index: 838612 (C4 (Web Text))

Max Activating Token Index: 499

Text #13

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2195. Min Act: -0.0000

Data Index: 355292 (C4 (Web Text))

Max Activating Token Index: 270

Text #14

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2274. Min Act: -0.0000

Data Index: 1187736 (C4 (Web Text))

Max Activating Token Index: 484

Text #15

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2068. Min Act: -0.0000

Data Index: 977527 (C4 (Web Text))

Max Activating Token Index: 197

Text #16

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2327. Min Act: -0.0000

Data Index: 439238 (C4 (Web Text))

Max Activating Token Index: 114

Text #17

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2097. Min Act: -0.0000

Data Index: 518847 (C4 (Web Text))

Max Activating Token Index: 579

Text #18

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2223. Min Act: -0.0000

Data Index: 692583 (C4 (Web Text))

Max Activating Token Index: 1014

Text #19

Max Range: 0.2941. Min Range: -0.2941

Max Act: 0.2037. Min Act: -0.0000

Data Index: 18121 (C4 (Web Text))

Max Activating Token Index: 908