Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5225 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.4805. Min Act: -0.0000

Data Index: 448550 (C4 (Web Text))

Max Activating Token Index: 207

Text #1

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.2672. Min Act: -0.0000

Data Index: 1289502 (C4 (Web Text))

Max Activating Token Index: 676

Text #2

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.4977. Min Act: -0.0000

Data Index: 900083 (C4 (Web Text))

Max Activating Token Index: 306

Text #3

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.2968. Min Act: -0.0000

Data Index: 1212651 (C4 (Web Text))

Max Activating Token Index: 621

Text #4

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.1135. Min Act: -0.0000

Data Index: 708187 (C4 (Web Text))

Max Activating Token Index: 242

Text #5

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.1377. Min Act: -0.0000

Data Index: 945674 (C4 (Web Text))

Max Activating Token Index: 354

Text #6

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.9881. Min Act: -0.0000

Data Index: 498435 (C4 (Web Text))

Max Activating Token Index: 152

Text #7

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.9207. Min Act: -0.0000

Data Index: 1124000 (C4 (Web Text))

Max Activating Token Index: 21

Text #8

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.6949. Min Act: -0.0000

Data Index: 614001 (C4 (Web Text))

Max Activating Token Index: 784

Text #9

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7986. Min Act: -0.0000

Data Index: 978958 (C4 (Web Text))

Max Activating Token Index: 391

Text #10

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.8927. Min Act: -0.0000

Data Index: 328686 (C4 (Web Text))

Max Activating Token Index: 919

Text #11

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7394. Min Act: -0.0000

Data Index: 1275434 (C4 (Web Text))

Max Activating Token Index: 359

Text #12

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7081. Min Act: -0.0000

Data Index: 1118181 (C4 (Web Text))

Max Activating Token Index: 613

Text #13

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7898. Min Act: -0.0000

Data Index: 719377 (C4 (Web Text))

Max Activating Token Index: 916

Text #14

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7763. Min Act: -0.0000

Data Index: 947309 (C4 (Web Text))

Max Activating Token Index: 597

Text #15

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7556. Min Act: -0.0000

Data Index: 254697 (C4 (Web Text))

Max Activating Token Index: 194

Text #16

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.5598. Min Act: -0.0000

Data Index: 799999 (C4 (Web Text))

Max Activating Token Index: 239

Text #17

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.7216. Min Act: -0.0000

Data Index: 714802 (C4 (Web Text))

Max Activating Token Index: 504

Text #18

Max Range: 2.4977. Min Range: -2.4977

Max Act: 2.0068. Min Act: -0.0000

Data Index: 59756 (C4 (Web Text))

Max Activating Token Index: 776

Text #19

Max Range: 2.4977. Min Range: -2.4977

Max Act: 1.4186. Min Act: -0.0000

Data Index: 483801 (C4 (Web Text))

Max Activating Token Index: 350