Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4717 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.2168. Min Act: -0.0000

Data Index: 1506034 (Python Code)

Max Activating Token Index: 768

Text #1

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.1231. Min Act: -0.0000

Data Index: 1602264 (Python Code)

Max Activating Token Index: 674

Text #2

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.1731. Min Act: -0.0000

Data Index: 1387968 (Python Code)

Max Activating Token Index: 890

Text #3

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.0784. Min Act: -0.0000

Data Index: 1470562 (Python Code)

Max Activating Token Index: 687

Text #4

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.1503. Min Act: -0.0000

Data Index: 1516009 (Python Code)

Max Activating Token Index: 365

Text #5

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.1382. Min Act: -0.0000

Data Index: 159230 (C4 (Web Text))

Max Activating Token Index: 109

Text #6

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.1716. Min Act: -0.0000

Data Index: 1018346 (C4 (Web Text))

Max Activating Token Index: 129

Text #7

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.0390. Min Act: -0.0000

Data Index: 1451712 (Python Code)

Max Activating Token Index: 991

Text #8

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9546. Min Act: -0.0000

Data Index: 85905 (C4 (Web Text))

Max Activating Token Index: 202

Text #9

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9727. Min Act: -0.0000

Data Index: 1536675 (Python Code)

Max Activating Token Index: 741

Text #10

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.0896. Min Act: -0.0000

Data Index: 1025476 (C4 (Web Text))

Max Activating Token Index: 806

Text #11

Max Range: 1.2168. Min Range: -1.2168

Max Act: 1.0044. Min Act: -0.0000

Data Index: 1579360 (Python Code)

Max Activating Token Index: 518

Text #12

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9770. Min Act: -0.0000

Data Index: 1398535 (Python Code)

Max Activating Token Index: 64

Text #13

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9162. Min Act: -0.0000

Data Index: 1487044 (Python Code)

Max Activating Token Index: 809

Text #14

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9592. Min Act: -0.0000

Data Index: 1577886 (Python Code)

Max Activating Token Index: 316

Text #15

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9284. Min Act: -0.0000

Data Index: 51846 (C4 (Web Text))

Max Activating Token Index: 449

Text #16

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9446. Min Act: -0.0000

Data Index: 1548051 (Python Code)

Max Activating Token Index: 696

Text #17

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.8922. Min Act: -0.0000

Data Index: 139629 (C4 (Web Text))

Max Activating Token Index: 287

Text #18

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9061. Min Act: -0.0000

Data Index: 97597 (C4 (Web Text))

Max Activating Token Index: 655

Text #19

Max Range: 1.2168. Min Range: -1.2168

Max Act: 0.9129. Min Act: -0.0000

Data Index: 1623420 (Python Code)

Max Activating Token Index: 704