Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3211 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.2328. Min Act: -0.0000

Data Index: 336399 (C4 (Web Text))

Max Activating Token Index: 730

Text #1

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0705. Min Act: -0.0000

Data Index: 802474 (C4 (Web Text))

Max Activating Token Index: 363

Text #2

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9884. Min Act: -0.0000

Data Index: 1231256 (C4 (Web Text))

Max Activating Token Index: 925

Text #3

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9794. Min Act: -0.0000

Data Index: 749599 (C4 (Web Text))

Max Activating Token Index: 784

Text #4

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9735. Min Act: -0.0000

Data Index: 747829 (C4 (Web Text))

Max Activating Token Index: 684

Text #5

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0534. Min Act: -0.0000

Data Index: 811911 (C4 (Web Text))

Max Activating Token Index: 104

Text #6

Max Range: 1.2328. Min Range: -1.2328

Max Act: 1.0158. Min Act: -0.0000

Data Index: 516667 (C4 (Web Text))

Max Activating Token Index: 307

Text #7

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8925. Min Act: -0.0000

Data Index: 374250 (C4 (Web Text))

Max Activating Token Index: 403

Text #8

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9158. Min Act: -0.0000

Data Index: 800921 (C4 (Web Text))

Max Activating Token Index: 494

Text #9

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.9854. Min Act: -0.0000

Data Index: 332372 (C4 (Web Text))

Max Activating Token Index: 78

Text #10

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8666. Min Act: -0.0000

Data Index: 313264 (C4 (Web Text))

Max Activating Token Index: 510

Text #11

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8909. Min Act: -0.0000

Data Index: 187782 (C4 (Web Text))

Max Activating Token Index: 729

Text #12

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8886. Min Act: -0.0000

Data Index: 937913 (C4 (Web Text))

Max Activating Token Index: 976

Text #13

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8016. Min Act: -0.0000

Data Index: 977226 (C4 (Web Text))

Max Activating Token Index: 888

Text #14

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.7179. Min Act: -0.0000

Data Index: 218896 (C4 (Web Text))

Max Activating Token Index: 720

Text #15

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8702. Min Act: -0.0000

Data Index: 1025754 (C4 (Web Text))

Max Activating Token Index: 946

Text #16

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8042. Min Act: -0.0000

Data Index: 957001 (C4 (Web Text))

Max Activating Token Index: 494

Text #17

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.8765. Min Act: -0.0000

Data Index: 898828 (C4 (Web Text))

Max Activating Token Index: 903

Text #18

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.7973. Min Act: -0.0000

Data Index: 1337435 (C4 (Web Text))

Max Activating Token Index: 518

Text #19

Max Range: 1.2328. Min Range: -1.2328

Max Act: 0.7774. Min Act: -0.0000

Data Index: 1023347 (C4 (Web Text))

Max Activating Token Index: 635