Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 900 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.5187. Min Act: -0.0000

Data Index: 1239470 (C4 (Web Text))

Max Activating Token Index: 404

Text #1

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.5209. Min Act: -0.0000

Data Index: 685486 (C4 (Web Text))

Max Activating Token Index: 495

Text #2

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.5793. Min Act: -0.0000

Data Index: 926036 (C4 (Web Text))

Max Activating Token Index: 639

Text #3

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.4000. Min Act: -0.0000

Data Index: 412223 (C4 (Web Text))

Max Activating Token Index: 745

Text #4

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.2477. Min Act: -0.0000

Data Index: 1346940 (C4 (Web Text))

Max Activating Token Index: 671

Text #5

Max Range: 2.5793. Min Range: -2.5793

Max Act: 2.2234. Min Act: -0.0000

Data Index: 1328048 (C4 (Web Text))

Max Activating Token Index: 642

Text #6

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.5276. Min Act: -0.0000

Data Index: 1009507 (C4 (Web Text))

Max Activating Token Index: 313

Text #7

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.5055. Min Act: -0.0000

Data Index: 1131726 (C4 (Web Text))

Max Activating Token Index: 1015

Text #8

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.2457. Min Act: -0.0000

Data Index: 573239 (C4 (Web Text))

Max Activating Token Index: 772

Text #9

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.3445. Min Act: 0.0000

Data Index: 1091674 (C4 (Web Text))

Max Activating Token Index: 997

Text #10

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.4288. Min Act: -0.0000

Data Index: 743992 (C4 (Web Text))

Max Activating Token Index: 454

Text #11

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.3006. Min Act: -0.0000

Data Index: 878517 (C4 (Web Text))

Max Activating Token Index: 871

Text #12

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.2993. Min Act: -0.0000

Data Index: 167439 (C4 (Web Text))

Max Activating Token Index: 944

Text #13

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.1933. Min Act: -0.0000

Data Index: 1305048 (C4 (Web Text))

Max Activating Token Index: 954

Text #14

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.3381. Min Act: -0.0000

Data Index: 106910 (C4 (Web Text))

Max Activating Token Index: 752

Text #15

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.1683. Min Act: -0.0000

Data Index: 223575 (C4 (Web Text))

Max Activating Token Index: 306

Text #16

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.1459. Min Act: -0.0000

Data Index: 597166 (C4 (Web Text))

Max Activating Token Index: 603

Text #17

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.1222. Min Act: -0.0000

Data Index: 150322 (C4 (Web Text))

Max Activating Token Index: 495

Text #18

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.0836. Min Act: -0.0000

Data Index: 1336597 (C4 (Web Text))

Max Activating Token Index: 381

Text #19

Max Range: 2.5793. Min Range: -2.5793

Max Act: 1.0871. Min Act: -0.0000

Data Index: 1082489 (C4 (Web Text))

Max Activating Token Index: 726