Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2996 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.7505. Min Act: -0.0001

Data Index: 548715 (C4 (Web Text))

Max Activating Token Index: 875

Text #1

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.7024. Min Act: -0.0001

Data Index: 296089 (C4 (Web Text))

Max Activating Token Index: 978

Text #2

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6877. Min Act: -0.0001

Data Index: 721175 (C4 (Web Text))

Max Activating Token Index: 552

Text #3

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6682. Min Act: -0.0001

Data Index: 8905 (C4 (Web Text))

Max Activating Token Index: 765

Text #4

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6857. Min Act: -0.0001

Data Index: 439335 (C4 (Web Text))

Max Activating Token Index: 980

Text #5

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6462. Min Act: -0.0001

Data Index: 1061072 (C4 (Web Text))

Max Activating Token Index: 177

Text #6

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6658. Min Act: -0.0001

Data Index: 227797 (C4 (Web Text))

Max Activating Token Index: 221

Text #7

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6359. Min Act: -0.0001

Data Index: 944673 (C4 (Web Text))

Max Activating Token Index: 160

Text #8

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6295. Min Act: -0.0001

Data Index: 1149568 (C4 (Web Text))

Max Activating Token Index: 684

Text #9

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6341. Min Act: -0.0001

Data Index: 248005 (C4 (Web Text))

Max Activating Token Index: 262

Text #10

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6135. Min Act: -0.0001

Data Index: 378057 (C4 (Web Text))

Max Activating Token Index: 389

Text #11

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6362. Min Act: -0.0001

Data Index: 1123374 (C4 (Web Text))

Max Activating Token Index: 81

Text #12

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6109. Min Act: -0.0001

Data Index: 575256 (C4 (Web Text))

Max Activating Token Index: 267

Text #13

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5832. Min Act: -0.0001

Data Index: 286660 (C4 (Web Text))

Max Activating Token Index: 142

Text #14

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.6161. Min Act: -0.0001

Data Index: 1250710 (C4 (Web Text))

Max Activating Token Index: 415

Text #15

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5933. Min Act: -0.0001

Data Index: 462407 (C4 (Web Text))

Max Activating Token Index: 207

Text #16

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5867. Min Act: -0.0001

Data Index: 103243 (C4 (Web Text))

Max Activating Token Index: 354

Text #17

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5673. Min Act: -0.0001

Data Index: 85146 (C4 (Web Text))

Max Activating Token Index: 669

Text #18

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5792. Min Act: -0.0001

Data Index: 446685 (C4 (Web Text))

Max Activating Token Index: 666

Text #19

Max Range: 0.7505. Min Range: -0.7505

Max Act: 0.5964. Min Act: -0.0001

Data Index: 219579 (C4 (Web Text))

Max Activating Token Index: 241