Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5513 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1510. Min Act: -0.0000

Data Index: 1589193 (Python Code)

Max Activating Token Index: 451

Text #1

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1431. Min Act: -0.0000

Data Index: 1447157 (Python Code)

Max Activating Token Index: 731

Text #2

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1296. Min Act: -0.0000

Data Index: 1573111 (Python Code)

Max Activating Token Index: 638

Text #3

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1320. Min Act: -0.0000

Data Index: 1555546 (Python Code)

Max Activating Token Index: 494

Text #4

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1409. Min Act: -0.0000

Data Index: 421800 (C4 (Web Text))

Max Activating Token Index: 830

Text #5

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1269. Min Act: -0.0000

Data Index: 1441248 (Python Code)

Max Activating Token Index: 872

Text #6

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1399. Min Act: -0.0000

Data Index: 1035712 (C4 (Web Text))

Max Activating Token Index: 708

Text #7

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1241. Min Act: -0.0000

Data Index: 1535674 (Python Code)

Max Activating Token Index: 684

Text #8

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1155. Min Act: -0.0000

Data Index: 575347 (C4 (Web Text))

Max Activating Token Index: 676

Text #9

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1240. Min Act: -0.0000

Data Index: 1422122 (Python Code)

Max Activating Token Index: 592

Text #10

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1239. Min Act: -0.0000

Data Index: 1411785 (Python Code)

Max Activating Token Index: 663

Text #11

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1105. Min Act: -0.0000

Data Index: 1257884 (C4 (Web Text))

Max Activating Token Index: 1010

Text #12

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1147. Min Act: -0.0000

Data Index: 1622355 (Python Code)

Max Activating Token Index: 71

Text #13

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1217. Min Act: -0.0000

Data Index: 1492847 (Python Code)

Max Activating Token Index: 917

Text #14

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1167. Min Act: -0.0000

Data Index: 1624502 (Python Code)

Max Activating Token Index: 321

Text #15

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1164. Min Act: -0.0000

Data Index: 1579798 (Python Code)

Max Activating Token Index: 664

Text #16

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1203. Min Act: -0.0000

Data Index: 1449535 (Python Code)

Max Activating Token Index: 762

Text #17

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1122. Min Act: -0.0000

Data Index: 1493994 (Python Code)

Max Activating Token Index: 796

Text #18

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1151. Min Act: -0.0000

Data Index: 1460562 (Python Code)

Max Activating Token Index: 938

Text #19

Max Range: 0.1510. Min Range: -0.1510

Max Act: 0.1142. Min Act: -0.0000

Data Index: 1587344 (Python Code)

Max Activating Token Index: 831