Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5623 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.6100. Min Act: -0.0000

Data Index: 1462954 (Python Code)

Max Activating Token Index: 806

Text #1

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.6032. Min Act: -0.0000

Data Index: 1598360 (Python Code)

Max Activating Token Index: 546

Text #2

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.5730. Min Act: -0.0000

Data Index: 1550612 (Python Code)

Max Activating Token Index: 414

Text #3

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4618. Min Act: -0.0000

Data Index: 1522395 (Python Code)

Max Activating Token Index: 457

Text #4

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4148. Min Act: -0.0000

Data Index: 1420852 (Python Code)

Max Activating Token Index: 984

Text #5

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3312. Min Act: -0.0000

Data Index: 1454930 (Python Code)

Max Activating Token Index: 909

Text #6

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4366. Min Act: -0.0000

Data Index: 1516125 (Python Code)

Max Activating Token Index: 661

Text #7

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4765. Min Act: -0.0000

Data Index: 1469114 (Python Code)

Max Activating Token Index: 633

Text #8

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4049. Min Act: -0.0000

Data Index: 1418577 (Python Code)

Max Activating Token Index: 835

Text #9

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3604. Min Act: -0.0000

Data Index: 1463937 (Python Code)

Max Activating Token Index: 657

Text #10

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3738. Min Act: -0.0000

Data Index: 1419557 (Python Code)

Max Activating Token Index: 975

Text #11

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.4341. Min Act: -0.0000

Data Index: 1443863 (Python Code)

Max Activating Token Index: 351

Text #12

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3818. Min Act: -0.0000

Data Index: 1380331 (Python Code)

Max Activating Token Index: 516

Text #13

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3384. Min Act: -0.0000

Data Index: 1472753 (Python Code)

Max Activating Token Index: 815

Text #14

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3959. Min Act: -0.0000

Data Index: 1425039 (Python Code)

Max Activating Token Index: 351

Text #15

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3310. Min Act: -0.0000

Data Index: 1583183 (Python Code)

Max Activating Token Index: 998

Text #16

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3435. Min Act: -0.0000

Data Index: 1421618 (Python Code)

Max Activating Token Index: 433

Text #17

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.2622. Min Act: -0.0000

Data Index: 1489706 (Python Code)

Max Activating Token Index: 426

Text #18

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.3253. Min Act: -0.0000

Data Index: 1454451 (Python Code)

Max Activating Token Index: 237

Text #19

Max Range: 1.6100. Min Range: -1.6100

Max Act: 1.2311. Min Act: -0.0000

Data Index: 1457134 (Python Code)

Max Activating Token Index: 493