Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 623 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.2945. Min Range: -3.2945

Max Act: 3.1584. Min Act: -0.0000

Data Index: 1221949 (C4 (Web Text))

Max Activating Token Index: 495

Text #1

Max Range: 3.2945. Min Range: -3.2945

Max Act: 3.2945. Min Act: -0.0000

Data Index: 830724 (C4 (Web Text))

Max Activating Token Index: 708

Text #2

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.8679. Min Act: -0.0000

Data Index: 1223728 (C4 (Web Text))

Max Activating Token Index: 197

Text #3

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.8530. Min Act: -0.0000

Data Index: 794298 (C4 (Web Text))

Max Activating Token Index: 765

Text #4

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7856. Min Act: -0.0000

Data Index: 1261711 (C4 (Web Text))

Max Activating Token Index: 256

Text #5

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7747. Min Act: -0.0000

Data Index: 784217 (C4 (Web Text))

Max Activating Token Index: 908

Text #6

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7385. Min Act: -0.0000

Data Index: 1250734 (C4 (Web Text))

Max Activating Token Index: 777

Text #7

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7272. Min Act: -0.0000

Data Index: 1133545 (C4 (Web Text))

Max Activating Token Index: 940

Text #8

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7169. Min Act: -0.0000

Data Index: 615235 (C4 (Web Text))

Max Activating Token Index: 856

Text #9

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6974. Min Act: -0.0000

Data Index: 147521 (C4 (Web Text))

Max Activating Token Index: 964

Text #10

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6693. Min Act: -0.0000

Data Index: 250212 (C4 (Web Text))

Max Activating Token Index: 576

Text #11

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.8070. Min Act: -0.0000

Data Index: 791493 (C4 (Web Text))

Max Activating Token Index: 989

Text #12

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6170. Min Act: -0.0000

Data Index: 189064 (C4 (Web Text))

Max Activating Token Index: 391

Text #13

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6618. Min Act: -0.0000

Data Index: 69280 (C4 (Web Text))

Max Activating Token Index: 223

Text #14

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6338. Min Act: -0.0000

Data Index: 735050 (C4 (Web Text))

Max Activating Token Index: 142

Text #15

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6480. Min Act: -0.0000

Data Index: 15927 (C4 (Web Text))

Max Activating Token Index: 841

Text #16

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6232. Min Act: -0.0000

Data Index: 560956 (C4 (Web Text))

Max Activating Token Index: 955

Text #17

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.4874. Min Act: -0.0000

Data Index: 818352 (C4 (Web Text))

Max Activating Token Index: 1003

Text #18

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.6301. Min Act: -0.0000

Data Index: 258204 (C4 (Web Text))

Max Activating Token Index: 465

Text #19

Max Range: 3.2945. Min Range: -3.2945

Max Act: 2.7383. Min Act: -0.0000

Data Index: 429680 (C4 (Web Text))

Max Activating Token Index: 737