Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 979 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.3805. Min Act: -0.0001

Data Index: 1653810 (Python Code)

Max Activating Token Index: 882

Text #1

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.3115. Min Act: -0.0001

Data Index: 1504735 (Python Code)

Max Activating Token Index: 807

Text #2

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.3143. Min Act: -0.0001

Data Index: 1590529 (Python Code)

Max Activating Token Index: 576

Text #3

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.3250. Min Act: -0.0001

Data Index: 1498554 (Python Code)

Max Activating Token Index: 328

Text #4

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2882. Min Act: -0.0001

Data Index: 1613558 (Python Code)

Max Activating Token Index: 261

Text #5

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2749. Min Act: -0.0001

Data Index: 1631733 (Python Code)

Max Activating Token Index: 223

Text #6

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.3014. Min Act: -0.0001

Data Index: 1517804 (Python Code)

Max Activating Token Index: 298

Text #7

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2535. Min Act: -0.0001

Data Index: 1580232 (Python Code)

Max Activating Token Index: 238

Text #8

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2584. Min Act: -0.0001

Data Index: 1432657 (Python Code)

Max Activating Token Index: 852

Text #9

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2743. Min Act: -0.0001

Data Index: 1617616 (Python Code)

Max Activating Token Index: 348

Text #10

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2739. Min Act: -0.0001

Data Index: 1432498 (Python Code)

Max Activating Token Index: 331

Text #11

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2851. Min Act: -0.0001

Data Index: 1543446 (Python Code)

Max Activating Token Index: 202

Text #12

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2520. Min Act: -0.0001

Data Index: 1633910 (Python Code)

Max Activating Token Index: 981

Text #13

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2435. Min Act: -0.0001

Data Index: 1577819 (Python Code)

Max Activating Token Index: 984

Text #14

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2491. Min Act: -0.0001

Data Index: 1626413 (Python Code)

Max Activating Token Index: 769

Text #15

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2422. Min Act: -0.0001

Data Index: 1550552 (Python Code)

Max Activating Token Index: 228

Text #16

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2748. Min Act: -0.0001

Data Index: 1487733 (Python Code)

Max Activating Token Index: 604

Text #17

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2381. Min Act: -0.0001

Data Index: 1471247 (Python Code)

Max Activating Token Index: 309

Text #18

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2531. Min Act: -0.0001

Data Index: 1606051 (Python Code)

Max Activating Token Index: 628

Text #19

Max Range: 0.3805. Min Range: -0.3805

Max Act: 0.2687. Min Act: -0.0001

Data Index: 1463868 (Python Code)

Max Activating Token Index: 638